Improving the use of research evidence in guideline development: 16. Evaluation

BACKGROUND: The World Health Organization (WHO), like many other organisations around the world, has recognised the need to use more rigorous processes to ensure that health care recommendations are informed by the best available research evidence. This is the last of a series of 16 reviews that have been prepared as background for advice from the WHO Advisory Committee on Health Research to WHO on how to achieve this. OBJECTIVES: We reviewed the literature on evaluating guidelines and recommendations, including their quality, whether they are likely to be up-to-date, and their implementation. We also considered the role of guideline developers in undertaking evaluations that are needed to inform recommendations. METHODS: We searched PubMed and three databases of methodological studies for existing systematic reviews and relevant methodological research. We did not conduct systematic reviews ourselves. Our conclusions are based on the available evidence, consideration of what WHO and other organisations are doing and logical arguments. KEY QUESTIONS AND ANSWERS: Our answers to these questions were informed by a review of instruments for evaluating guidelines, several studies of the need for updating guidelines, discussions of the pros and cons of different research designs for evaluating the implementation of guidelines, and consideration of the use of uncertainties identified in systematic reviews to set research priorities. How should the quality of guidelines or recommendations be appraised? • WHO should put into place processes to ensure that both internal and external review of guidelines is undertaken routinely. • A checklist, such as the AGREE instrument, should be used. • The checklist should be adapted and tested to ensure that it is suitable to the broad range of recommendations that WHO produces, including public health and health policy recommendations, and that it includes questions about equity and other items that are particularly important for WHO guidelines. When should guidelines or recommendations be updated? • Processes should be put into place to ensure that guidelines are monitored routinely to determine if they are in need of updating. • People who are familiar with the topic, such as Cochrane review groups, should do focused, routine searches for new research that would require revision of the guideline. • Periodic review of guidelines by experts not involved in developing the guidelines should also be considered. • Consideration should be given to establishing guideline panels that are ongoing, to facilitate routine updating, with members serving fixed periods with a rotating membership. How should the impact of guidelines or recommendations be evaluated? • WHO headquarters and regional offices should support member states and those responsible for policy decisions and implementation to evaluate the impact of their decisions and actions by providing advice regarding impact assessment, practical support and coordination of efforts. • Before-after evaluations should be used cautiously and when there are important uncertainties regarding the effects of a policy or its implementation, randomised evaluations should be used when possible. What responsibility should WHO take for ensuring that important uncertainties are addressed by future research when the evidence needed to inform recommendations is lacking? • Guideline panels should routinely identify important uncertainties and research priorities. This source of potential priorities for research should be used systematically to inform priority-setting processes for global research

Improving the use of research evidence in guideline development: 11. Incorporating considerations of cost-effectiveness, affordability and resource implications

BACKGROUND: The World Health Organization (WHO), like many other organisations around the world, has recognised the need to use more rigorous processes to ensure that health care recommendations are informed by the best available research evidence. This is the 11(th )of a series of 16 reviews that have been prepared as background for advice from the WHO Advisory Committee on Health Research to WHO on how to achieve this. OBJECTIVES: We reviewed the literature on incorporating considerations of cost-effectiveness, affordability and resource implications in guidelines and recommendations. METHODS: We searched PubMed and three databases of methodological studies for existing systematic reviews and relevant methodological research. We did not conduct systematic reviews ourselves. Our conclusions are based on the available evidence, consideration of what WHO and other organisations are doing and logical arguments. KEY QUESTIONS AND ANSWERS: When is it important to incorporate cost-effectiveness, resource implications and affordability considerations in WHO guidelines (which topics)? • For cost-effectiveness: The need for cost/effectiveness information should be dictated by the specific question, of which several may be addressed in a single guideline. It is proposed that the indications for undertaking a cost-effectiveness analysis (CEA) could be a starting point for determining which recommendation(s) in the guideline would benefit from such analysis. • For resource implications/affordability: The resource implications of each individual recommendation need to be considered when implementation issues are being discussed. How can cost-effectiveness, resource implications and affordability be explicitly taken into account in WHO guidelines? • For cost-effectiveness: ∘ If data are available, the ideal time to consider cost-effectiveness is during the evidence gathering and synthesizing stage. However, because of the inconsistent availability of CEAs and the procedural difficulty associated with adjusting results from different CEAs to make them comparable, it is also possible for cost-effectiveness to be considered during the stage of developing recommendations. ∘ Depending on the quantity and quality and relevance of the data available, such data can be considered in a qualitative way or in a quantitative way, ranging from a listing of the costs to a modelling exercise. At the very least, a qualitative approach like a commentary outlining the economic issues that need to be considered is necessary. If a quantitative approach is to be used, the full model should be transparent and comprehensive. • For resource implications/affordability: ∘ Resource implications, including health system changes, for each recommendation in a WHO guideline should be explored. At the minimum, a qualitative description that can serve as a gross indicator of the amount of resources needed, relative to current practice, should be provided. How does one provide guidance in contextualizing guideline recommendations at the country level based on considerations of cost-effectiveness, resource implications and affordability? • All models should be made available and ideally are designed to allow for analysts to make changes in key parameters and reapply results in their own country. • In the global guidelines, scenarios and extensive sensitivity/uncertainty analysis can be applied. Resource implications for WHO • From the above, it is clear that guidelines development groups will need a health economist. There is need to ensure that this is included in the budget for guidelines and that there is in-house support for this as well

Clinical practice guidelines for the foot and ankle in rheumatoid arthritis: a critical appraisal

Background: Clinical practice guidelines are recommendations systematically developed to assist clinical decision-making and inform healthcare. In current rheumatoid arthritis (RA) guidelines, management of the foot and ankle is under-represented and the quality of recommendation is uncertain. This study aimed to identify and critically appraise clinical practice guidelines for foot and ankle management in RA. Methods: Guidelines were identified electronically and through hand searching. Search terms 'rheumatoid arthritis', 'clinical practice guidelines' and related synonyms were used. Critical appraisal and quality rating were conducted using the Appraisal of Guidelines for Research and Evaluation (AGREE) II instrument. Results: Twenty-four guidelines were included. Five guidelines were high quality and recommended for use. Five high quality and seven low quality guidelines were recommended for use with modifications. Seven guidelines were low quality and not recommended for use. Five early and twelve established RA guidelines were recommended for use. Only two guidelines were foot and ankle specific. Five recommendation domains were identified in both early and established RA guidelines. These were multidisciplinary team care, foot healthcare access, foot health assessment/review, orthoses/insoles/splints, and therapeutic footwear. Established RA guidelines also had an 'other foot care treatments' domain. Conclusions: Foot and ankle management for RA features in many clinical practice guidelines recommended for use. Unfortunately, supporting evidence in the guidelines is low quality. Agreement levels are predominantly 'expert opinion' or 'good clinical practice'. More research investigating foot and ankle management for RA is needed prior to inclusion in clinical practice guidelines

An educational game for teaching clinical practice guidelines to Internal Medicine residents: development, feasibility and acceptability

<p>Abstract</p> <p>Background</p> <p>Adherence to Clinical Practice Guidelines (CPGs) remains suboptimal among internal medicine trainees. Educational games are of growing interest and have the potential to improve adherence to CPGs. The objectives of this study were to develop an educational game to teach CPGs in Internal Medicine residency programs and to evaluate its feasibility and acceptability.</p> <p>Methods</p> <p>We developed the Guide-O-Game^©in the format of a TV game show with questions based on recommendations of CPGs. The development of the Guide-O-Game^©consisted of the creation of a multimedia interactive tool, the development of recommendation-based questions, and the definition of the game's rules. We evaluated its feasibility through pilot testing and its acceptability through a qualitative process.</p> <p>Results</p> <p>The multimedia interactive tool uses a Macromedia Flash web application and consists of a manager interface and a user interface. The user interface allows the choice of two game styles. We created so far 16 sets of questions relating to 9 CPGs. The pilot testing proved that the game was feasible. The qualitative evaluation showed that residents considered the game to be acceptable.</p> <p>Conclusion</p> <p>We developed an educational game to teach CPGs to Internal Medicine residents that is both feasible and acceptable. Future work should evaluate its impact on educational outcomes.</p

The systematic guideline review: method, rationale, and test on chronic heart failure

Background: Evidence-based guidelines have the potential to improve healthcare. However, their de-novo-development requires substantial resources-especially for complex conditions, and adaptation may be biased by contextually influenced recommendations in source guidelines. In this paper we describe a new approach to guideline development-the systematic guideline review method (SGR), and its application in the development of an evidence-based guideline for family physicians on chronic heart failure (CHF). Methods: A systematic search for guidelines was carried out. Evidence-based guidelines on CHF management in adults in ambulatory care published in English or German between the years 2000 and 2004 were included. Guidelines on acute or right heart failure were excluded. Eligibility was assessed by two reviewers, methodological quality of selected guidelines was appraised using the AGREE instrument, and a framework of relevant clinical questions for diagnostics and treatment was derived. Data were extracted into evidence tables, systematically compared by means of a consistency analysis and synthesized in a preliminary draft. Most relevant primary sources were re-assessed to verify the cited evidence. Evidence and recommendations were summarized in a draft guideline. Results: Of 16 included guidelines five were of good quality. A total of 35 recommendations were systematically compared: 25/35 were consistent, 9/35 inconsistent, and 1/35 un-rateable (derived from a single guideline). Of the 25 consistencies, 14 were based on consensus, seven on evidence and four differed in grading. Major inconsistencies were found in 3/9 of the inconsistent recommendations. We re-evaluated the evidence for 17 recommendations (evidence-based, differing evidence levels and minor inconsistencies) - the majority was congruent. Incongruity was found where the stated evidence could not be verified in the cited primary sources, or where the evaluation in the source guidelines focused on treatment benefits and underestimated the risks. The draft guideline was completed in 8.5 man-months. The main limitation to this study was the lack of a second reviewer. Conclusion: The systematic guideline review including framework development, consistency analysis and validation is an effective, valid, and resource saving-approach to the development of evidence-based guidelines

Improving the use of research evidence in guideline development: 14. Reporting guidelines

BACKGROUND: The World Health Organization (WHO), like many other organisations around the world, has recognised the need to use more rigorous processes to ensure that health care recommendations are informed by the best available research evidence. This is the 14(th )of a series of 16 reviews that have been prepared as background for advice from the WHO Advisory Committee on Health Research to WHO on how to achieve this. OBJECTIVES: We reviewed the literature on reporting guidelines and recommendations. METHODS: We searched PubMed and three databases of methodological studies for existing systematic reviews and relevant methodological research. We did not conduct systematic reviews ourselves. Our conclusions are based on the available evidence, consideration of what WHO and other organisations are doing and logical arguments. KEY QUESTIONS AND ANSWERS: There is little empirical evidence that addresses these questions. Our answers are based on logical arguments and standards put forward by other groups. What standard types of recommendations or reports should WHO use? • WHO should develop standard formats for reporting recommendations to facilitate recognition and use by decision makers for whom the recommendations are intended, and to ensure that all the information needed to judge the quality of a guideline, determine its applicability and, if needed, adapt it, is reported. • WHO should develop standard formats for full systematically developed guidelines that are sponsored by WHO, rapid assessments, and guidelines that are endorsed by WHO. • All three formats should include the same information as full guidelines, indicating explicitly what the group preparing the guideline did not do, as well as the methods that were used. • These formats should be used across clinical, public health and health systems recommendations. How should recommendations be formulated and reported? • Reports should be structured, using headings that correspond to those suggested by the Conference on Guideline Standardization or similar headings. • The quality of evidence and strength of recommendations should be reported explicitly using a standard approach. • The way in which recommendations are formulated should be adapted to the specific characteristics of a specific guideline. • Urgent attention should be given to developing a template that provides decision makers with the relevant global evidence that is needed to inform a decision and offers practical methods for incorporating the context specific evidence and judgements that are needed

Clinical practice guidelines within the Southern African development community: a descriptive study of the quality of guideline development and concordance with best evidence for five priority diseases

<p>Abstract</p> <p>Background</p> <p>Reducing the burden of disease relies on availability of evidence-based clinical practice guidelines (CPGs). There is limited data on availability, quality and content of guidelines within the Southern African Development Community (SADC). This evaluation aims to address this gap in knowledge and provide recommendations for regional guideline development.</p> <p>Methods</p> <p>We prioritised five diseases: HIV in adults, malaria in children and adults, pre-eclampsia, diarrhoea in children and hypertension in primary care. A comprehensive electronic search to locate guidelines was conducted between June and October 2010 and augmented with email contact with SADC Ministries of Health. Independent reviewers used the AGREE II tool to score six quality domains reporting the guideline development process. Alignment of the evidence-base of the guidelines was evaluated by comparing their content with key recommendations from accepted reference guidelines, identified with a content expert, and percentage scores were calculated.</p> <p>Findings</p> <p>We identified 30 guidelines from 13 countries, publication dates ranging from 2003-2010. Overall the '<it>scope and purpose' </it>and '<it>clarity and presentation' </it>domains of the AGREE II instrument scored highest, median 58%(range 19-92) and 83%(range 17-100) respectively. '<it>Stakeholder involvement' </it>followed with median 39%(range 6-75). '<it>Applicability'</it>, '<it>rigour of development' </it>and '<it>editorial independence' </it>scored poorly, all below 25%. Alignment with evidence was variable across member states, the lowest scores occurring in older guidelines or where the guideline being evaluated was part of broader primary healthcare CPG rather than a disease-specific guideline.</p> <p>Conclusion</p> <p>This review identified quality gaps and variable alignment with best evidence in available guidelines within SADC for five priority diseases. Future guideline development processes within SADC should better adhere to global reporting norms requiring broader consultation of stakeholders and transparency of process. A regional guideline support committee could harness local capacity to support context appropriate guideline development.</p

Preconception care of women with diabetes: a review of current guideline recommendations

<p>Abstract</p> <p>Background</p> <p>The prevalence of type 2 diabetes mellitus (T2DM) continues to rise worldwide. More women from developing countries who are in the reproductive age group have diabetes resulting in more pregnancies complicated by T2DM, and placing both mother and foetus at higher risk. Management of these risks is best achieved through comprehensive preconception care and glycaemic control, both prior to, and during pregnancy. The aim of this review was to compare the quality and content of current guidelines concerned with the preconception care of women with diabetes and to develop a summary of recommendations to assist in the management of diabetic women contemplating pregnancy.</p> <p>Methods</p> <p>Relevant clinical guidelines were identified through a search of several databases (MEDLINE, SCOPUS and The Cochrane Library) and relevant websites. Five guidelines were identified. Each guideline was assessed for quality using the AGREE instrument. Guideline recommendations were extracted, compared and contrasted.</p> <p>Results</p> <p>All guidelines were assessed as being of high quality and strongly recommended for use in practice. All were consistent in counselling about the risk of congenital malformation related to uncontrolled blood sugar preconceptionally, ensuring adequate contraception until glycaemic control is achieved, use of HBA1C to monitor metabolic control, when to commence insulin and switching from ACE inhibitors to other antihypertensives. Major differences were in the targets recommended for optimal metabolic control and opinion regarding the usage of metformin as an adjunct or alternative treatment before or during pregnancy.</p> <p>Conclusions</p> <p>International guidelines for the care of women with diabetes who are contemplating pregnancy are consistent in their recommendations; however some are more comprehensive than others. Having established current standards for the preconception care of diabetic women, there is now a need to focus on guideline implementation through an examination of the barriers and enablers to successful implementation, and the applicability of the recommendations in the local setting.</p

Etiologic workup in cases of cryptogenic stroke: protocol for a systematic review and comparison of international clinical practice guidelines

Background Stroke is a leading cause of death and disability worldwide. Identifying the aetiology of ischaemic stroke is essential in order to initiate appropriate and timely secondary prevention measures to reduce the risk of recurrence. For the majority of ischaemic strokes, the aetiology can be readily identified, but in at least 30% of cases the exact aetiology cannot be determined using existing investigative protocols. Such strokes are classed as ‘cryptogenic’ or as a stroke of unknown origin. However, there exists substantial variation in clinical practice when investigating cases of seemingly cryptogenic stroke, often reflecting local service availability and the preferences of treating clinicians. This variation in practice is compounded by the lack of international consensus as to the optimum level and timing of investigations required following a stroke. To address this gap, we aim to systematically review and compare recommendations in evidence-based clinical practice guidelines (CPGs) that relate to the assessment and investigation of the aetiology of ischaemic stroke, and any subsequent diagnosis of cryptogenic stroke. Method We will search for CPGs using electronic databases (MEDLINE, Health Management Information Consortium (HMIC), EMBASE, and CINAHL); relevant websites and search engines (e.g. guideline specific websites, governmental, charitable, professional practice organisations); and hand-searching of bibliographies and reference lists. Two reviewers will independently screen titles, abstracts and CPGs using a pre-defined relevance criteria form. From each included CPG we will extract definitions and terms for cryptogenic stroke; recommendations related to assessment and investigation of the aetiology of stroke, including grade of recommendations and underpinning evidence. Quality of the included CPGs will be assessed using the AGREE II (Appraisal of Guidelines for Research & Evaluation) tool. Recommendations across the CPGs will be summarised descriptively highlighting areas of convergence and divergence between CPGs. Discussion To our knowledge this will be the first review to systematically compare recommendations of international CPGs on investigating the aetiology of ischaemic stroke. The findings will allow for a better understanding of international perspectives on the optimum level of investigations required following a stroke, and thus contribute to achieving greater international consensus on best practice in this important and complex area

Synthesising practice guidelines for the development of community-based exercise programmes after stroke

This is a freely-available open access publication. Please cite the published version which is available via the DOI link in this record.Multiple guidelines are often available to inform practice in complex interventions. Guidance implementation may be facilitated if it is tailored to particular clinical issues and contexts. It should also aim to specify all elements of interventions that may mediate and modify effectiveness, including both their content and delivery. We conducted a focused synthesis of recommendations from stroke practice guidelines to produce a structured and comprehensive account to facilitate the development of community-based exercise programmes after stroke.National Institute for Health Research (NIHR) Collaboration for Leadership in Applied Health Research and Care (CLAHRC) for the South West Peninsul